<html>
<!-- <link href="../enzo.css" rel="stylesheet" type="text/css"> -->
  <head>
    <title>Starting up with Enzo</title>
  </head>
<body> 
    <h1>Starting up with Enzo</h1>
    <p>This is a step-by-step guide to getting Enzo compiled and a test problem run on your computer of choice. 
      We assume no previous experience with the code.  The only assumption is that the computer that you are installing
      Enzo on has <a href="http://www-unix.mcs.anl.gov/mpi/">MPI</a> and 
      <a href="http://hdf.ncsa.uiuc.edu/hdf4.html">HDF 4</a> or 
      <a href="http://hdf.ncsa.uiuc.edu/HDF5">HDF 5</a>
      (or possibly both) installed on your machine.  Also, we assume that your machine has
      <a href="http://make.paulandlesley.org/">GNU Make</a> installed.</p>

    <p>
      This page shows actual inputs and outputs from an Enzo installation performed on Titan <tt>(titan.ncsa.uiuc.edu)</tt>,
      the <a href="http://www.ncsa.uiuc.edu">NCSA</a> IA-64 Linux cluster.  Click 
      <a href="http://www.ncsa.uiuc.edu/UserInfo/Resources/Hardware/IA64LinuxCluster/">here</a> for more
      information about that machine.  Though we show inputs from this machine, they should be generally applicable
      to your machine of choice, as long as it runs some unix variant.  We have successfully compiled and run Enzo
      on Sun, SGI, IBM, Compaq, Hewlett-Packard
      and Cray machines, as well as Apple computers running OS X and a wide variety of machines
      running the Linux operating system.
    </p>


    <a name="obtain"></a> 
    <h2>Obtaining the Enzo package</h2>
    <p>You can download the Enzo package from the Enzo web site, 
      <a href="http://cosmos.ucsd.edu/enzo/">http://cosmos.usd.edu/enzo/</a>.  Click on 'download' and then download the 
      enzo code bundle from the website.  You should now have a file called <tt>enzo-1.0.0.tar.gz</tt> on your
      computer.  Copy this file to the machine that you want to run Enzo on.  You then untar this by typing:</p>

    <p><tt>user02:~101% gunzip enzo-1.0.0.tar.gz; tar xvf enzo-1.0.0.tar</tt></p>
    
    <p>The file will untar into a directory called <tt>enzo-1.0.0</tt>, and the source code should occupy about
      6 megabytes.  You need a total of roughly 100 megabytes to compile Enzo and all of its associated utilities.</p>


    <a name="compile"></a>
    <h2>Compiling Enzo</h2>
    <p>Once you untar the code bundle, go to the enzo top level directory and run the 
      configuration tool.  You do this by typing:</p>

    <p>
      <tt>user02:~102% cd enzo-1.0.0</tt><br>

      <tt>user02:~103% ./configure</tt>
    </p>

    <p>The GNU configuration tool will then determine what kind of system you have, your compilers, etc.  
      There will be quite a bit of output.  At the end, the tool will print out several lines of information that 
      looks something like this:</p>

<pre>
   Architecture:        Intel Itanium
   Machine:             Generic
   MPI:                 On
   jbPerf/jbMem:        Off
   I/O:                 HDF5
   Global precision:    Double
   Particles precision: Double
   Optimization level:  Debug + Optimize
   Object file type:    64-bit objects
   FFT:                 F90 FFT
   Code optimizations:  On (excludes un-tested optimizations)
   Test suite hooks:    Off
</pre>

    <p>This is the configuration that Enzo is going to compile with.  In the case of the output above, 
      that means that Enzo will compile with MPI enabled (to allow you to perform simulations using
      multiple processors), HDF 5 as the IO format, double precision variables throughout, and optimization
      and debugging code enabled.  There are a large number of possible configurations, which can be seen by
      typing:</p>

    <p><tt>user02:~104% make help-config</tt></p>

    <p>Which will print out the various options, which we do not show here here for the sake of brevity.  At
    any time, you can see the current configuration of the code by typing:</p>


    <p><tt>user02:~105% make show-config</tt></p>

    <p>If your machine is not one of the supported platforms (the various NCSA and SDSC computers) you may have to 
      manually set paths to libraries and compilers.  Instructions on how to do this are in the file 
      <tt>README.config</tt> in the Enzo top level directory.  This process should be relatively painless
      assuming that all of the libraries needed for Enzo are installed correctly.  If you have problems
      finding specific libraries, consult your system administrator.</p>

    <p>After you have finished setting all of your options, the configuration code will have generated makefiles
      in every subdirectory with the appropriate compiler and library information.  To actually compile the code 
      (assuming you have set the various options how you want them) you then type:</p>

    <p><tt>user02:~106% make</tt></p>

    <p>Assuming that the configuration code has found everything correctly, you will then be rewarded with thousands 
      of lines of output as Enzo and its tools are compiled. When the compilation has finished (assuming there are
      no errors), you should then type:</p>

    <p><tt>user02:~107% make install</tt></p>

    <p>Which will then put all of the executables in the <tt>enzo-1.0.0/bin</tt> directory.</p>

    <p>If you need to recompile the code, you should make sure to clean out all of the subdirectories.
      You do this by typing:</p>

    <p><tt>user02:~108% make clean</tt></p>

    <p>This will remove all of the binaries and object files.  If you are having problems with compilation
      you may need
      to do a more thorough cleaning, which removes all of the configuration files and machine-generated
      makefiles as well as the executables and object files.  You do this by typing:</p>

    <p><tt>user02:~109% make distclean</tt></p>
    
    <p>You will then have to start the compilation instructions from the beginning of this section.</p>

    <a name="runtest"></a>
    <h2>Running an Enzo test problem</h2>
    <p>This section assumes that you have compiled enzo with mpi <b>OFF</b>.  You do this by typing
      <tt>make mpi-off</tt> before actually compiling the code by typing 
      <tt>make</tt> and <tt>make install</tt>.</p>

    <p>
      At this point you should have a compiled version of enzo in the directory <tt>enzo-1.0.0/bin</tt>.
      You can run a test problem by changing to the directory <tt>enzo-1.0.0/amr_mpi/exe</tt>.  If you
      type <tt>ls</tt>, you'll see there are several parameter files in this directory:
    </p>

<pre>
user02:~/enzo-1.0.0/amr_mpi/exe110% ls 
AdiabaticExpansion           ATOMIC.DAT            lookup_metal0.3.data  ShockTube            TestGravityMotion
AMRCosmologySimulation       CollapseTest          NFWTest               show_projection.pro  TestGravitySphere
AMRShockPool2D               CollideTest           PressurelessCollapse  SphericalInfall      TREECOOL
AMRShockTube                 cool_rates.in         ProjectionParameters  StripTest            viewit_extract*
AMRZeldovichPancake          ExtremeAdvectionTest  ShockPool2D           SupernovaeRestart    WavePool
AnalyzeClusterParameterFile  GravityStripTest      ShockPool3D           TestGravity          ZeldovichPancake
</pre>

    <p>Most of these files are parameter files for various test problems.  See 
      <a href="../amr_guide/index-enzo.html#Test%20Problem%20Parameters">this section of the Enzo Guide</a>
      for more information on test problems and their parameters.  We're going to run the Shock Tube
      test.  This is a one-dimensional problem which is known as the Sod Shock Tube and is commonly used to 
      test various aspects of a hydro code.  We'll be using the parameter file <tt>ShockTube</tt>.  Examine the 
      contents of this file by typing:

    <p><tt>user02:~/enzo-1.0.0/amr_mpi/exe111% more ShockTube</tt></p>

    <p>The following parameters will then be displayed:</p>

<pre>
#
# AMR PROBLEM DEFINITION FILE: ShockTube test (1 grid version)
#
#  define problem
#
ProblemType            = 1       // Shock Tube
TopGridRank            = 1
TopGridDimensions      = 100
HydroMethod            = 0
#
#  set I/O and stop/start parameters
#
StopTime               = 0.251
dtDataDump             = 0.4
#
#  set hydro parameters
#
Gamma                  = 1.4
PPMDiffusionParameter  = 0       // diffusion off
CourantSafetyNumber    = 0.8
#
#  set grid refinement parameters
#
StaticHierarchy           = 1    // static hierarchy
MaximumRefinementLevel    = 1    // use up to 2 levels
RefineBy                  = 4    // refinement factor
MinimumSlopeForRefinement = 0.2  // set this to &lt;= 0.2 to refine CD
# 
#  set some global parameters 
#
tiny_number            = 1.0e-6  // fixes velocity slope problem 
MinimumEfficiency      = 0.4     // better value for 1d than 0.2
</pre>

    <p>You then can run the test problem by typing:</p>

    <p><tt>user02:~/enzo-1.0.0/amr_mpi/exe112% ../../bin/enzo ShockTube</tt></p>

    <p>This will produce some output to the screen and several files whose names start with <tt>data0000</tt> and
      <tt>data0001</tt>.  You can get more verbose output from enzo
      (for debugging purposes) by using the <tt>-d</tt> flag, ie, use <tt>enzo -d</tt> instead of
      simply <tt>enzo</tt>.  If you do that the output maybe be difficult to follow, so you may wish to 
      redirect the output of enzo into a log file, like this:</p>

    <p><tt>user02:~/enzo-1.0.0/amr_mpi/exe113% ../../bin/enzo -d ShockTube &gt; shocktubetest.log</tt></p>

    <p>And then you can examine the contents of the log file by typing <tt>more shocktubetest.log</tt>.</p>

    <p>The output of this program are two sets of files whose names start with <tt>data0000</tt> and
      <tt>data0001</tt>.  The first set is the simulation output at the beginning of the simulation, and 
      the second set is the final output.  Each set of files has one file with no extension, which is the 
      restart parameter file.  This contains text-based information that Enzo needs in order to restart the
      simulation, if you wanted to do that.  The files that end in the <tt>.hierarchy</tt> extension are text-based
      information on the Enzo grid hierarchy.  This particular test is unigrid, so the hierarchy file has 
      a single (albeit multi-line) entry.  Typically AMR simulations have anywhere between dozens and tens
      of thousands of entries in this file.  The files that end in <tt>.boundary</tt> and <tt>.boundary.hdf</tt> contain
      information on the boundary conditions for the test problem, and are outside the scope of this discussion.
      The files that end in <tt>.grid0001</tt> contain the actual simulation outputs.  These can be examined using the
      <tt>h5ls</tt> and <tt>h5dump</tt> utilities, which come with your <a href="http://hdf.ncsa.uiuc.edu/HDF5">HDF 5</a>
      library in the bin directory.  <tt>h5ls</tt> displays the data sets that are stored in a given HDF 5 file, like so:</p>

<pre>
user02:~/enzo-1.0.0/amr_mpi/exe114%  h5ls data0001.grid0001 
Density                  Dataset {100}
Total\ Energy            Dataset {100}
x-velocity               Dataset {100}
user02:~/enzo-1.0.0/amr_mpi/exe115% 
</pre>

    <p>The <tt>h5dump</tt> command actually dumps out the values of a given dataset.  If we want to view the x-velocity
      dataset that we know is in the file <tt>data0001.grid0001</tt>, we do this:</p>

<p><tt>user02:~/enzo-1.0.0/amr_mpi/exe145% h5dump -d x-velocity data0001.grid0001</tt></p>

    <p>Which outputs a great deal of information about the x-velocity dataset.  <tt>h5ls</tt> and <tt>h5dump</tt> are
      very useful tools for looking at HDF 5 files and doing quick checks of the data.  For more advanced visualization
      and data analysis techniques, you may wish to refer to the Cookbook sections on 
      <a href="analysis_cook_top.html">data analysis</a> and <a href="dataviz_top.html">data visualization</a>, and the
      Enzo user's guide section on <a href="../amr_guide/analyze.html">data analysis</a>.</p>

    <p>Congratulations!  If you've been following along on your own, you've now run your first Enzo test problem and
      taken a look at the output.</p>

    <a name="runcos"></a>
    <h2>Running an Enzo cosmology simulation</h2>

    <p>In order to run an example cosmology simulation you will need to recompile the code with MPI <b>ON</b>.  Refer to
      the section above on <a href="#compile">compiling Enzo</a> to see how to do that.  After recompiling with MPI on, 
      you should create a directory to run the simulation in.  This is because Enzo cosmology simulations create quite 
      a few output files, so it's best to store them in their own directory.  For the purposes of this example I'm 
      going to assume that you have created a directory called <tt>EnzoTestSim</tt> in your home directory.  You should
      then download these two files:  <a href="Example_Cosmology_Sim.inits">Example_Cosmology_Sim.inits</a> and 
      <a href="Example_Cosmology_Sim.param">Example_Cosmology_Sim.param</a> which are the initial conditions parameter
      file and the simulation parameter file, respectively.  Put these two files in that directory, along with the enzo
      binary, the ring binary,  and the initial conditions generator binary (the binaries are named
      <tt>enzo</tt>, <tt>ring</tt> and <tt>inits</tt>, respectively).</p>


    <h3>Creating initial conditions</h3>

    <p>The first step in preparing the simulation is to create the initial conditions.  The file 
      <a href="Example_Cosmology_Sim.inits">Example_Cosmology_Sim.inits</a> is a text file which contains a list
      of parameter file names with their associated values.  These values tell the initial conditions generator
      useful information like the simulation box size, the cosmological parameters and the size of the root grid.
      The code then takes that information and creates a set of initial conditions.  <tt>inits</tt> is run
      by typing this command:</p>

    <p><tt>user02:~/EnzoTestSim150% ./inits -d Example_Cosmology_Sim.inits</tt></p>
      
    <p><tt>inits</tt> will produce some output to the screen to tell you what it is doing, and will write
      five files:  <tt>GridDensity</tt>, <tt>GridVelocities</tt>, <tt>ParticlePositions</tt>,
      <tt>ParticleVelocities</tt> and <tt>PowerSpectrum.out</tt>.  The first four files contain information
      on initial conditions for the baryon and dark matter componenets of the simulation, and are HDF 5 files.  
      The last file is an ascii file which contains information on the power spectrum used to generate the
      initial conditions.</p>

    <h3>Parallel IO - the ring tool</h3>

    <p>This simulation is quite small.  The root grid is only 32 cells on a side and we allow a maximum
      of three levels of mesh refinement.  Still, we will use the <tt>ring</tt> tool, since it is important
      for larger simulations of sizes typically used for doing science.</p>

    <p>The <tt>ring</tt> tool is part of the Enzo parallel IO (input-output) scheme.  Examine the last section of
      the <a href="Example_Cosmology_Sim.param">parameter file</a> for this example simulation and you will see:</p>

<pre>
#
# IO parameters
#
ParallelRootGridIO = 1
ParallelParticleIO = 1
</pre> 

    <p>These two parameters turn on parallel IO for both grids and particles.  In a serial IO simulation where multiple
      processors are being used, the master processor reads
      in all of the grid and particle initial condition information and parcels out portions of the data to the other processors.
      Similarly, all simulation output goes through the master processor as well.
      This is fine for relatively small simulations using only a few processors, but slows down the code considerably
      when a huge simulation is being run on hundreds of processors.  Turning on the parallel IO options allows each processor
      to perform its own IO, which greatly decreases the amount of time the code spends performing IO.</p>


    <p>The process for parallelizing grid and particle information is quite different. Since we know exactly where every 
      grid cell in a structured Eulerian grid is in space, and these cells are stored in a regular and predictable order
      in the initial conditions files, turning on ParallelRootGridIO simply tells each processor to
      figure out which portions of the arrays in the <tt>GridDensity</tt> and <tt>GridVelocities</tt> belong to it, and
      then read in only that part of the file.  The particle files (<tt>ParticlePositions</tt> and <tt>ParticleVelocities</tt>)
      store the particle information in no particular order, so in order to efficiently parallelize the particle IO the
      <tt>ring</tt> tool is used.  <tt>ring</tt> is run on the same number of processors as the simulation that you intend 
      to run, and can be used right before the simulation itself is run.  In <tt>ring</tt>, each processor reads in an 
      equal fraction of the particle position and velocity information into a list, flags the particles that belong in its
      simulation spatial domain, 
      and then passes its portion of the total list on to another processor.  After each portion of the list has made its
      way to every processor, each processor then collects all of the particle and velocity information that belongs to it
      and writes them out into files called <tt>PPos.nnnn</tt> and <tt>PVel.nnnn</tt>, where nnnn is the processor number.
      Turning on the <tt>ParallelParticleIO</tt> flag in the Enzo parameter file instructs Enzo to look for these files.</p>

    <p>For the purpose of this example, I'm going to run ring and Enzo on 4 processors, but as an interactive job on Titan.
      One enters interactive parallel mode by typing:</tt>

    <p><tt>user02:~/EnzoTestSim151%  qsub -I -V -l walltime=00:30:00,nodes=2:ppn=2:prod</tt></p>

    <p>This tells the machine that I want two compute nodes with two processors on each node (for a
      total of 4 processors) for a half hour of wall clock time.  See
      <a href="http://www.ncsa.uiuc.edu/UserInfo/Resources/Hardware/IA64LinuxCluster/Doc/Jobs.html#qsub_I">this web page</a>
      for more information on running interactive jobs on Titan.  You may have to wait a bit until nodes become available, and
      then you will probably be left in your home directory, so you will have to change back to the EnzoTestSim subdirectory.
      You then run ring on the particle files by typing:</p>

    <p><tt>user02:~/EnzoTestSim154%  vmirun ./ring ParticlePositions ParticleVelocities</tt></p>

    <p>This will then produce some output to your screen, and will generate 8 files: 
      <tt>PPos.0000</tt> through <tt>PPos.0003</tt> and 
      <tt>PVel.0000</tt> through <tt>PVel.0003</tt>.  Note that 
      if you are using a different machine or platform, you may use something other than vmirun for MPI-parallel
      applications. Consult your system administrator or system documentation for more information.</p>

    <p>Congratulations, you're now ready to run your cosmology simulation!</p>

    <h3>Running an Enzo cosmology simulation</h3>

    <p>After all of this preparation, running the simulation itself should be straightforward.  You 
      start enzo by typing:</p>

    <p><tt>user02:~/EnzoTestSim155%  vmirun ./enzo -d Example_Cosmology_Sim.param &gt; output.log</tt></p>

    <p>The simulation will now run.  The <tt>-d</tt> flag ensures a great deal of output, so we redirect it
      into a log file called <tt>output.log</tt> for later examination.  This particular simulation should take
      approximately ten minutes to run on 4 processors on Titan.  When the simulation is done, enzo will
      display the message <tt>"Successful run, exiting."</tt></p>

    <p>Enzo is a complicated code, with a similarly complicated output format.  See the Enzo User Guide page on 
      <a href="../amr_guide/output.html">the Enzo output format</a> for more information on the data outputs.

    <p>Congratulations!  If you've made it this far, you have now successfully run a simulation using Enzo!</p>
    
<p>&nbsp;</p>
<p>
<a href="index.html">Previous - Index</a><br>
<a href="machine_top.html">Next - Running Enzo on specific machines</a><br>
</p>

<p>&nbsp;</p>
<p>
<a href="../index.html">Go to the Enzo home page</a>
</p>
<hr WIDTH="100%">
<center>&copy; 2004 &nbsp; <a href="http://cosmos.ucsd.edu">Laboratory for Computational Astrophysics</a><br></center>
<center>last modified February 2004<br>
by <a href="mailto:bwoshea (AT) lanl.gov">B.W. O'Shea</a></center>


</body>
</html>
